Promoting Repeatability Through Open Runs
نویسندگان
چکیده
The 2015 Text REtrieval Conference (TREC) introduced the concept of ‘Open Runs’ in response to the increasing focus on repeatability of information retrieval experiments. An Open Run is a TREC submission backed by a software repository such that the software in the repository reproduces the system that created that exact run. The ID of the repository was captured during the process of submitting the run and published as part of the metadata describing the run in the TREC proceedings. Submitting a run as an Open Run was optional: either a repository ID was provided at submission time or it was not, and further processing of the run was identical in either case. Unfortunately, this initial offering was not successful. While a healthy 79 runs were submitted as Open Runs, we could not in fact reproduce any of them. This paper explores possible reasons for the difficulties and makes suggestions for how to address the deficiencies so as to strengthen the Open Run program for TREC 2016.
منابع مشابه
Repeatability of Detecting Visual Cortex Activity in Functional Magnetic Resonance Imaging
Introduction As functional magnetic resonance imaging (fMRI) is too expensive and time consuming, its frequent implementation is difficult. The aim of this study is to evaluate repeatability of detecting visual cortex activity in fMRI. Materials and Methods In this study, 15 normal volunteers (10 female, 5 male; Mean age±SD: 24.7±3.8 years) attended. Functional magnetic resonance images were ob...
متن کاملPromoting the Sense of Place Attachment through Enplaning the Meaning of Place in the Open Space of Aseman-e-Tabriz Residential Complex
The meaning of place and, consequently, the quality of relationship between people and places has become significant due to the current ever rising identity crises and the breakdown of meaning as a result of various phenomena such as globalization. Since the open space has lost its value and since the open space design is not considered in residential complexe...
متن کاملMultiple-breath washout measurements can be significantly shortened in children.
Multiple-breath washout (MBW)-derived lung clearance index (LCI) is a sensitive measure of ventilation inhomogeneity in patients with cystic fibrosis (CF), but LCI measurement is time consuming. We systematically assessed ways to shorten LCI measurements. In 68 school-aged children (44 with mild CF lung disease) three standard nitrogen (N2) MBWs were applied. We assessed repeatability and diagn...
متن کاملUrban Open Spaces Supporting Physical Activity and Promoting Citizen’s Health: A Systematic Review
Background and Objective: Due to the limited individual approach to behavior change, health promotion researchers use community-based initiatives to understand the factors affecting physical activity and promote the health of citizens. Urban open spaces can facilitate participation in physical activity and the health of citizens. The aim of this study is to identify the indicators, attributes, ...
متن کاملRepeatability of Real World Training Experiments: A Case Study
We present a case study of reinforcement learning on a real robot, and we show how the need to perform a large number of experimental learning runs on a real robot and in a systematic way has required to redesign some of the robot hardware. We describe in detail the design of the robot and present results of the learning algorithm.
متن کامل